Automatically suggesting regions for blur kernel estimation

ABSTRACT

Embodiments of the present invention provide systems, methods, and computer storage media directed towards automatic selection of regions for blur kernel estimation. In one embodiment, a process divides a blurred image into a regions. From these regions a first region and a second region can be selected based on a number of edge orientations within the selected regions. A first blur kernel can then be estimated based on the first region and a second blur kernel can be estimated for the second region. The first and second blur kernel can then be utilized to respectively deblur a first and second portion of the image to produce a deblurred image. Other embodiments may be described and/or claimed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Ser. No. 15/161,073, filed onMay 20, 2016; which application is a continuation of U.S. Ser. No.14/061,131, filed on Oct. 23, 2013, both of which are herebyincorporated by reference in their entireties.

TECHNICAL FIELD

The subject matter disclosed herein generally relates to imagedeblurring and, more specifically, to automatically suggesting at leastone region for blur kernel estimation.

BACKGROUND

Taking handheld photos in low-light conditions is challenging. Sinceless light is available, longer exposure times are needed to obtain anadequately exposed image. A consequence of longer exposure times is thatthe quality of the photo is more susceptible to camera movement which,in turn, results in a blurred image. In order to avoid camera movement,photographers frequently use a tripod to support a camera. However, useof a tripod to remove movement is not always practical. This isparticularly relevant with the proliferation of mobile phones withbuilt-in cameras. Consequently, in many, if not almost allcircumstances, camera shake is likely to be an issue resulting in blurrypictures.

Blur may be removed, with varying degrees of success, from a blurredimage in an attempt to recover a latent sharp image. Known techniquesutilize a single blur kernel to deblur an entire image usingdeconvolution. These known techniques require user input to identify aregion in the blurred image from which to estimate a blur kernel.Further, known techniques are very computationally intensive and high inmemory usage.

It will thus be appreciated that it is challenging for casual users,without trained eyes, to select an appropriate region for blur kernelestimation. This, and long computation times, can become a frustratingprocess for users trying to deblur an image. Accordingly, providingassistance to a user in selecting a region for blur kernel estimationwould be very beneficial.

BRIEF DESCRIPTION OF THE DRAWINGS

The present technology is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings.

FIG. 1A is a diagram of a network environment, in accordance with anexample embodiment, in which applications for editing images may bedeployed.

FIG. 1B is a block diagram of a user device, in accordance with anexample embodiment, configured to edit images.

FIG. 2 is a block diagram of example components of a blur kernel sizingengine, in accordance with an example embodiment.

FIG. 3 is a flow diagram of a method, in accordance with an exampleembodiment, for determining a suggested size of a blur kernel for ablurred region in an image.

FIG. 4 is a flow diagram of a method, in accordance with an exampleembodiment, for estimating a size of a blur kernel for a blurred regionin an image using an autocorrelation map.

FIG. 5A shows an example blurred image.

FIG. 5B shows an example latent sharp image for the blurred image ofFIG. 5A.

FIG. 5C shows an example blur kernel for the blurred image of FIG. 5A.

FIG. 6A shows a screen shot illustrating an autocorrelation map, inaccordance with an example embodiment.

FIG. 6B shows a screen shot illustrating selected connected componentswithin the example autocorrelation map of FIG. 6A.

FIG. 7 is a flow diagram of a method, in accordance with an exampleembodiment, for estimating a size of a blur kernel for a blurred imageusing a latent image process.

FIG. 8A depicts an example image of blur kernel estimation from a latentimage.

FIG. 8B depicts the measurement of connected components within the blurkernel estimation of FIG. 8A.

FIG. 9 is a block diagram illustrating example components of a regionsuggestion engine, in accordance with an example embodiment.

FIG. 10 shows a screen shot of suggested regions, in accordance with anexample embodiment, for automatically estimating a blur kernel.

FIG. 11 is a flow diagram of a method, in accordance with an exampleembodiment, for automatically selecting a region in an image forestimating a blur kernel.

FIG. 12 is an example blurred image showing positioning of a blur kernelat a selected region of the blurred image.

FIG. 13 is a block diagram of example components of an image deblurringengine, in accordance with an example embodiment, to reduce spatiallyvarying blur using multiple blur kernels.

FIG. 14 is a flow diagram of a method, in accordance with an exampleembodiment, for deblurring a blurred image.

FIG. 15 is an example blurred image showing positioning of multiple blurkernels, in accordance with an example embodiment, at multiple blurredregions of the image.

FIG. 16A shows a blending region, in accordance with an exampleembodiment, between a first region, occupied by a first blur kernel, anda second region occupied by a second blur kernel, wherein the first andsecond regions overlap.

FIG. 16B is a schematic diagram illustrating a tile-based blendingarrangement, in accordance with an example embodiment.

FIG. 17 shows a graphical user interface (GUI), in accordance with anexample embodiment, for editing images.

FIG. 18 is a block diagram of example components of a display interface,in accordance with an example embodiment.

FIG. 19 is a flow diagram of a method, in accordance with an exampleembodiment, for presenting user editing controls for deblurring animage.

FIGS. 20A-20C are exploded views of a blur kernel zone of the GUI ofFIG. 17.

FIG. 21 is a GUI, in accordance with an example embodiment, illustratingan overlay of a blur kernel on an associated image.

FIG. 22 is a block diagram illustrating components of a machine,according to some example embodiments, configured to read instructionsfrom a machine-readable medium and perform any one or more of themethodologies described herein.

DETAILED DESCRIPTION

Overview

In the following description, for purposes of explanation, numerousspecific details are set forth to provide a thorough understanding ofexample embodiments. It will be evident to one skilled in the art,however, that the present subject matter may be practiced without thesespecific details.

Example methods and systems to automatically, without user input,suggest at least one region for estimation of a blur kernel areprovided. For example, a photo editing application (e.g., Adobe®Photoshop®, LightRoom® or the like) may access a blurred image anddefine a size for each blurred region (e.g., based on a scaling factor).Thereafter, metrics for the plurality of regions are determined, whereinthe metrics are based on a number of edge orientations within eachregion. A region from the plurality of regions is then selected based onthe determined metrics (e.g., a region having the largest number of edgeorientations). A blur kernel may then be estimated for the selectedregion and the image (e.g., all regions of the image) may then bedeblurred using the blur kernel. In some example embodiments, one ormore regions in which the number of edge orientations (e.g., of objectswithin a photograph) exceed a threshold value are selected forestimating the blur kernel. Optionally, the metrics are further based ona number of over-exposed or under-exposed pixels, wherein an influenceof the over-exposed or under-exposed pixels is weighted.

In an example embodiment, based on a comparison of the determinedmetrics and a threshold metric for selecting the region, the method mayfurther comprise modifying a size of each of the plurality of regions,determining metrics for each of the plurality of regions having amodified size, and selecting a region having the modified size that isassociated with a metric that satisfies the threshold metric forestimating the blur kernel.

Example Computing Environment

As described herein, in some example embodiments, systems and methodsare described that are configured to deblur images via an image or photoediting application, such as the Adobe® Photoshop® family ofapplications. The technology may be implemented by one or moreapplications resident on a computing device (e.g., mobile computingdevice) and/or in a networked environment (e.g., a cloud-based networkenvironment) where processing may, or may not, be distributed.

FIG. 1A is a diagram of a network environment 100, in accordance with anexample embodiment, for editing images or otherwise processing images(e.g., digital photographs). The network environment 100 is shown toinclude a user device 110 that supports a user interface 115 configuredto receive input from an associated user, and configured to presentinformation to the user to allow deblurring of images. The user device110 may include a web browser application, an application (e.g., “app”),or other programs configured to display images and/or allow the user toinput or identify images for editing. For example, the user interface115 may facilitate the access of one or more photographic images storedon the user device 110 (e.g., in a photo library), and import imagesfrom remote devices and/or applications and so on. The user interface115 may form part of a photo editing application (e.g., Adobe®Photoshop®, Light Room® or the like).

The user device 110 is shown to communicate over a network 120 with aserver 130 (e.g., a Software as a Service (SAAS) server), which providesa cloud-based and/or network-based image editing application 135. Theimage editing application 135 may communicate with the user device 110and its user interface 115 to facilitate the editing of images by theuser via the user device 110.

The network 120 may be any network that enables communication amongmachines, databases, and devices (mobile or otherwise). Accordingly, thenetwork 120 may be a wired network, a wireless network (e.g., a mobileor cellular network), a storage area network (SAN), or any suitablecombination thereof. In an example embodiment, the network 120 includesone or more portions of a private network, a public network (e.g., theInternet), or combination thereof. The user device 110 may be anysuitable computing device, such as a desktop computer, a laptopcomputer, a smart phone, a tablet, a gaming device, or any othercomputing device configured to facilitate interactions between a userand the image editing application 135.

It should be noted that the image editing application 135 may includevarious components, modules, and/or engines configured to facilitate thedeblurring, sharpening, and/or other modifications of images. Forexample, the images may have a single blurred region, a plurality ofblurred regions, and/or other artifacts that degrade the quality of theimages and hence removal thereof is desired.

In an example embodiment, the deblurring of an image (or portions of animage) includes techniques that attempt to estimate a latent sharp imagefor a blurred image (image including blurred regions) by convolving anestimated latent sharp image with a blur kernel. The blur kernel may berepresented by a convolution matrix. In an example embodiment,deblurring may utilize the following equation:b=l*k+n

-   -   where b is the blurred image, l is the latent sharp image, k is        a blur kernel, n is image noise, and * is a convolution operator        (as discussed in more detail below with reference to FIGS.        5A-C). Deconvolution of the blurred image b, which is a reverse        operation of the convolution operation, leads to the latent        sharp image l. Deconvolution, therefore, is an operation that        may reverse, change and/or modify blur, distortions, or other        imperfections in an image. Thus, solving for l by deconvolving        the blurred image b with the blur kernel k may lead to an        enhanced or sharper image where blurring is at least reduced in        blurred regions of the image.

The image editing application 135, therefore, may comprise variouscomponents to deblur images including, but not limited to, an imagedeblurring engine 140, a blur kernel sizing engine 150, and a regionsuggestion engine 160. The image deblurring engine 140 may be configuredto deblur images having multiple blurred regions (e.g., spatiallyvarying blur), the blur kernel sizing engine 150 may be configured toautomatically (e.g., without human intervention) determine a size of oneor more blur kernels utilized in deblurring images, and the regionsuggestion engine 160 may be configured to automatically identify and/orsuggest regions within a blurred image at which to apply and/or generateblur kernels.

Although the example image editing application 135 is shown as beingpart of a network environment 100, it may be deployed wholly, or inpart, on any of the machines, databases, or devices shown in FIG. 1A.For example, FIG. 1B is a block diagram of a user device 170, inaccordance with an example embodiment, configured to edit images. Theuser device 170 may be the same or similar to the user device 110 andmay be configured to support the image editing application 135 (as astandalone or networked device). For example, the user device 170 maystore and execute software/instructions to facilitate interactionsbetween a user and the image editing application 135 via the userinterface 115 of the user device 110.

In some example embodiments, the image editing application 135 mayperform, or cause to be performed, various image editing tasks atdifferent networked locations. For example, the image editingapplication 135 may remotely estimate blur kernels at the server 130,while locally deblurring images at the device 110, or vice versa. Theuse of different resources may be user-selectable, such that a user mayconfigure the image editing application 135 to perform certain tasksremotely, or in the cloud, and other tasks locally.

Examples of Automatically Determining the Size of a Blur Kernel

As described herein, deblurring an image involves deconvolving one ormore blurred regions using at least one blur kernel. At times, usersattempting to deblur an image having blurred regions may be unable (orof insufficient skill) to define an appropriate size of the blur kernelused to deblur the image. For example, when a selected size of the blurkernel is too small, the selected blur kernel may be considered anunsuitable kernel, which may not properly or sufficiently deblur theimage. Also, when a selected size of the blur kernel is too large, theselected blur kernel may not properly or sufficiently deblur the image.Therefore, automatically estimating and/or determining the size of blurkernels may provide users with suitable blur kernels of appropriate sizeto adequately deblur images (e.g., via existing deblurring algorithmsand/or the algorithms described herein). In an example embodiment, theautomated estimation of the size of the blur kernel is calculatedelectronically without human input influencing the size of the blurkernel. In another example embodiment, the size of the blur kernel mayat least partially be determined based on algorithms that do not dependon user input.

As described herein, example systems and methods are described that mayautomatically estimate or otherwise determine the size of a blur kernelused to deblur at least one region of a blurred image. FIG. 2 is a blockdiagram of example components (or modules) of a blur kernel sizingengine 150, in accordance with an example embodiment. The examplecomponents may be hardware, software, or a combination of hardware andsoftware, and may be executed by one or more processors. In the exampleembodiment, the blur kernel sizing engine 150 is shown to include animage module 210 and a blur kernel sizing module 220. The blur kernelsizing module 220 may include an autocorrelation-based size estimator222 to estimate a first size of the blur kernel, a latent image-basedsize estimator 224 to estimate a second size of the blur kernel, andprocess the results to suggest or recommend a size of the blur kernel tobe used to deblur the image. It is however to be noted that theautocorrelation-based size estimator 222 and the latent image-based sizeestimator 224 are merely examples of estimators, and that othermathematical techniques may be used in other embodiments.

The image module 210 may be configured (e.g., a programmed processor) toaccess an image (e.g., an image stored in a photo library on a device)having at least one blurred region. In an example embodiment, a userselects or identifies an image including a region to be deblurred usinga graphical user interface, and the image module 210 then accesses theimage (e.g., accesses data defining the image) for processing.

In some example embodiments, the blur kernel sizing module 220 isconfigured to estimate a size of a blur kernel for the accessed imageusing two different sizing estimators. For example, the blur kernelsizing module 220 may automatically determine a first value for a firstsize of a blur kernel for a blurred region using a first sizingestimator (e.g., the autocorrelation-based size estimator 222),automatically determine a second value for a second size of the blurkernel using a second, different, size estimator, and select a suggestedsize for the blur kernel that is based on the first value and the secondvalue, as described in more detail with reference to FIG. 3. It shouldbe appreciated that the estimated values for the sizes of the blurkernel may however be computed by the same estimator or any one or moreestimators.

FIG. 3 is a flow diagram of a method 300, in accordance with an exampleembodiment, for determining a suggested size of a blur kernel fordeblurring a blurred region in an image. The method 300 may be performedby the blur kernel sizing engine 150 and, accordingly, is describedherein merely by way of example with reference thereto. It will beappreciated that the method 300 may be performed on any suitablehardware.

In the method 300, the blur kernel sizing engine 150 may access datarepresenting an image (herein also referred to as accessing the image)having at least one blurred region (see operation 310). For example, theimage module 210 may access a blurred image including one or moreblurred regions. In some example embodiments, the image module 210 maydownsample the accessed image in order to reduce noise and/or reduce thesize of the blurred regions that are used to estimate the size of a blurkernel.

Thereafter, as shown in operations 320 and 330, the blur kernel sizingengine 150 automatically, without user input, determines a first valuefor a first size of a blur kernel, and automatically, without userinput, determines a second value for the size of the blur kernel to beused in deblurring a region of the image. In an example embodiment, twodifferent sizing techniques may be used to determine the first andsecond values. As mentioned above, the first size estimator may be anautocorrelation-based size estimator (e.g., the autocorrelation-basedsize estimator 222), which determines an estimated size of the blurkernel based on autocorrelating the accessed image with a derivative ofthe accessed image, as discussed by way of example below with referenceto FIG. 4.

As shown in operation 340, the method 300 may then automaticallydetermine a suggested size for the blur kernel based on the first andsecond values.

FIG. 4 is a flow diagram of a method 400, in accordance with an exampleembodiment, for estimating a size of a blur kernel for a blurred regionin an image using an autocorrelation process. The method 400 may beperformed by the autocorrelation-based size estimator 222 and,accordingly, is described herein merely by way of example with referencethereto. It will be appreciated that the method 400 may be performed onany suitable hardware.

A correlation operation f∘g between two images f and g may be definedas:f∘g=f**g

-   -   where ∘ is a correlation operator, and f* is a flipped version        of f, which may be flipped both horizontally and vertically        (e.g., f(x,y)=f(−x,−y) where x and y are pixel coordinates).        Autocorrelation occurs when f and g are identical. In some        cases, derivatives of natural images are usually weakly        correlated to one another, and autocorrelation results of image        derivatives may be approximated by a delta function, which has a        single nonzero peak, while other values are zero. However, when        an image is blurred and includes blurred regions, derivatives of        the image become more correlated to the image and/or to one        another, and the autocorrelation becomes less similar to a delta        function. Accordingly, in an example embodiment, a corresponding        blur kernel may be based the deviation of the autocorrelation        (e.g., delta function).

Mathematically, the autocorrelation of a derivative of a latent sharpimage l should be close to a delta function, such as:(d*l)∘(d*l)=((d*l)**(d*l))≈δ

-   -   where d is a discrete differential operator, and δ is a delta        function. However, the autocorrelation of a derivative of a        blurred image b is:        (d*b)∘(d*b)=(d*k*l)**(d*k*l)=(k**k)*((d*l)**(d*l))≈(k**k)*δ

The autocorrelation of (d*b) is related to the blur kernel k. Therefore,the actual blur size may be related to the nonzero elements in theautocorrelation result. For example, assuming a blur kernel size isgeneralized as W_(k)×H_(k), the size of k**k is (2W_(k)−1)×(2H_(k)−1).

Accordingly, in an example embodiment, in operation 410 theautocorrelation-based size estimator 222 calculates a derivative for theblurred regions of the accessed image. The autocorrelation-based sizeestimator 222 may apply a differential operator to data representing theaccessed image, such as a discrete differential operator d, defined as:

$d = \begin{bmatrix}0 & {- 1} & 1 \\0 & 1 & {- 1} \\0 & 0 & 0\end{bmatrix}$

FIG. 5A shows an example of a blurred image 510, FIG. 5B shows a latentsharp image 505 corresponding to the blurred image 510, and acorresponding blur kernel 515 is shown in FIG. 5C. Theautocorrelation-based size estimator 222 may calculate variousderivatives of the blurred image 510, such as first order derivative, asecond order derivative, and so on. By computing derivatives, in anexample embodiment, pixels in flat regions can be removed from beingconsidered in the autocorrelation map. Other derivative filters mayprovide similar effects, and they may be used instead.

Referring back to FIG. 4, in operation 420, the autocorrelation-basedsize estimator 222 may determine an autocorrelation map for thederivative. The autocorrelation map may reflect the autocorrelationbetween the blurred image 510 and the derivative of the blurred image510. In an example embodiment, the autocorrelation-based size estimator222 may compute the autocorrelation using a Fourier transform, or othersimilar technique.

Thereafter the method 400, as shown in operation 430, applies athreshold filter to the autocorrelation of the blurred image and thederivative thereof. For example, in order to determine the size of theblur kernel from the autocorrelation map, the autocorrelation-based sizeestimator 222 may identify the extent of nonzero elements within theautocorrelation map, and determine the size of the blur kernel based onthe nonzero elements (e.g., as described with reference to FIGS. 6A andB below). However, in some cases, noise and/or repetitive patterns, suchas textures in the latent image l, may introduce misleading nonzerovalues, which may cause the autocorrelation result to differ from adelta function. The autocorrelation-based size estimator 222 may,therefore, apply a threshold filter to the autocorrelation result toremove small non-zero components (e.g., less than a threshold value)from the autocorrelation result, in order to correctly and/or robustlyestimate the extent of nonzero elements caused by the blur kernel k,among other things.

In operation 440, a connected component in the autocorrelation map maybe identified (e.g., using the autocorrelation-based size estimator222). For example, the autocorrelation-based size estimator 222 maymeasure a cluster of nonzero elements within the autocorrelation map,which may then define the connected components.

FIG. 6A shows a screen shot 600 illustrating an example autocorrelationmap 605 determined using the method 400. As shown in FIG. 6A, thecomponents are symmetrical. FIG. 6B shows a screen shot 610 illustratingselected connected components 615 within the example autocorrelation map605. For example, the autocorrelation-based size estimator 222 mayidentify the connected component 615 of nonzero elements centered withinan autocorrelation map 605, and measure the dimensions (e.g., a maximumwidth and height) of the connected component 615 in a connectedcomponent zone, shown as box 620. The autocorrelation-based sizeestimator 222 may then determine and/or estimate the size of the blurkernel based on the measured width and height of the connected component615. For example, the autocorrelation-based size estimator 222 maydetermine the box 620 is 21 by 21 pixels, and automatically estimate thesize (e.g., determine a lower and/or higher value) of the blur kernel tobe 21 by 21 pixels, or within a range of sizes that includes thedetermined size of the box 620. It should be noted that the area or zoneof connected components may, in other embodiments, be of differentshapes and that a box is merely an example of such a shape.

In some example embodiments, the second size estimator (e.g., the latentimage-based size estimator 224) estimates a size based on an estimatedlatent image. For example, the latent image-based size estimator 224 mayautomatically generate and/or determine a rough guess or estimate of alatent image I for the blurred image, and use the estimated latent imageto automatically determine the size of the blur kernel.

FIG. 7 is a flow diagram of a method 700, in accordance with an exampleembodiment, for estimating a size of a blur kernel for an image using alatent image process. The method 700 may be performed by the latentimage-based size estimator 224 and, accordingly, is described hereinmerely by way of example with reference thereto. It will be appreciatedthat the method 700 may be performed on any suitable hardware.

In operation 710, the latent image-based size estimator 224 determinesan estimated latent image for the accessed image. The latent image-basedsize estimator 224 may apply a shock filter to a blurred image b torestore sharp edges within the image. The latent image-based sizeestimator 224 may then compute one or more gradient maps from the shockfiltered result. Optionally, gradient values of small magnitudes (e.g.,below or equal to a gradient threshold) may be removed to remove noiseand determine a gradient map p_(x) and p_(y), which represents theestimated latent image.

Thereafter, as shown in operation 720, a blur kernel may be estimatedfrom the latent image, for example, using the latent image-based sizeestimator 224. Given p_(x) and p_(y) are suitable for an estimatedlatent image, the method 700 may estimate (e.g., using the latentimage-based size estimator 224) a blur kernel k by solving the followingexample optimization equation:

$k = {\arg\underset{k}{\;\min}\left( {{{{p_{x}*k} - {d_{x}*b}}}^{2} + {{{p_{y}*k} - {d_{y}*b}}}^{2} + {\alpha{{d_{x}*k}}^{2}} + {\alpha{{d_{y}*k}}^{2}} + {\beta{k}^{2}}} \right)}$

-   -   where d_(x) and d_(y) are the first order discrete differential        operators along the x and y axes, the first two terms are data        fidelity terms based on image gradients, the third and fourth        terms are regularization terms to cause a blur kernel k to be        smooth and continuous, the fifth term is a Tikhonov        regularization term to further stabilize the optimization        problem, and α and β are regularization strength for each        regularization term.

In an example embodiment, the latent image-based size estimator 224solves the above equation using Fourier transforms, as follows:

${K\left( {u,v} \right)} = \frac{{\overset{\_}{P_{x}\left( {u,v} \right)}{D_{x}\left( {u,v} \right)}{B\left( {u,v} \right)}} + {\overset{\_}{P_{y}\left( {u,v} \right)}{D_{y}\left( {u,v} \right)}{B\left( {u,v} \right)}}}{{{P_{x}\left( {u,v} \right)}}^{2} + {{P_{y}\left( {u,v} \right)}}^{2} + {\alpha\left( {{{D_{x}\left( {u,v} \right)}}^{2} + {{D_{y}\left( {u,v} \right)}}^{2}} \right)} + \beta}$

-   -   where K, P_(x), P_(y), D_(x), D_(y) and B are Fourier transforms        of k, p_(x), p_(y), d_(x), d_(y) and b, respectively, and        P_(x)(u,v) is the complex conjugate of P_(x)(u,v). By applying        an inverse Fourier transform to K, an estimate of the blur        kernel k may be generated (e.g., by the latent image-based size        estimator 224). FIG. 8A depicts an example image 800 of a blur        kernel 810 estimated from the estimated latent image, including        nonzero elements 815 (see operation 710 in FIG. 7.).

Returning to the method 700, in operation 730, nonzero elements of theestimated blur kernel may be measured. For example, the latentimage-based size estimator 224 may measure the nonzero elements 815depicted in the blur kernel 810 by using various techniques describedherein, such as the techniques described with respect to FIG. 4. Asmentioned above, the size of the blur kernel may be determined from thenonzero elements 815.

For example, FIG. 8B depicts the measurement of connected components 825within an estimated blur kernel 820. A measured zone (connectedcomponent region) illustrated by a box 827 reflects measurement of theconnected components 825. For example, the box 827 may have a width of35 pixels and a height of 31 pixels, and the latent image-based sizeestimator 224 may automatically estimate the size (e.g., determine alower and/or higher value) of the blur kernel to be 35 by 31 pixels, orwithin a range of sizes that includes the determined size of the box827.

In some example embodiments, the blur kernel sizing engine 150 mayreceive input from a user which identifies an additional estimated sizefor the blur kernel. Thus, in an example embodiment, estimation of thesize of a blur kernel may be based on one or more automated estimationsand a user-defined estimation. These estimations may then, for example,be weighted and used to determine the blur kernel size that will be usedto deblur the image selected by the user for processing.

Accordingly, in some example embodiments, the blur kernel sizing engine150 may utilize the autocorrelation-based size estimator 222 to estimatea lower or smaller value of the size of the blur kernel (operation 320),and may utilize the latent image-based size estimator 224 to estimate ahigher or upper value of the size of the blur kernel (operation 330). Insome alternate embodiments, the blur kernel sizing engine 150 mayutilize the latent image-based size estimator 224 to estimate the loweror smaller value of the size of the blur kernel (operation 320), and mayutilize the autocorrelation-based size estimator 222 to estimate thehigher or upper value of the size of the blur kernel (operation 330).

Returning to operation 340 in FIG. 3, the blur kernel sizing engine 150may determine the suggested size for the blur kernel based on the firstvalue and the second value. For example, the blur kernel sizing engine150 may perform a variety of different determinations or follow variousselection criteria or rules when determining the suggested, or actual,size for the blur kernel. Example selection criteria or rules mayinclude:

Select the first value as the suggested size for the blur kernel whenthe value determined by the autocorrelation-based size estimator 222 islower than the value determined by the latent image-based size estimator224;

Select the first value as the suggested size for the blur kernel whenthe value determined by the autocorrelation-based size estimator 222 ishigher than the value determined by the latent image-based sizeestimator 224;

Select the second value as the suggested size for the blur kernel whenthe value determined by the autocorrelation-based size estimator 222 islower than the value determined by the latent image-based size estimator224;

Calculate the average of the first value and the second value as thesuggested size for the blur kernel;

Calculate the average of the first value, the second value, and a userdefined size as the suggested size for the blur kernel;

Calculate the median of the first value, the second value, and a userdefined size (and/or other estimated values) as the suggested size forthe blur kernel; or

Select the value estimated by the autocorrelation-based size estimator222 when the value is lower than the value estimated by the latentimage-based size estimator 224; and so on.

Thus, in some example embodiments, the blur kernel sizing engine 150 andthe methods 300 and 700 utilize one or more size estimation techniquesto automatically, with or without user input, determine the size of ablur kernel for an input blurred image.

Examples of Automatically Suggesting Regions for Blur Kernel Estimation

As described herein, deblurring an image may involve applying a blurkernel to a blurred region of an image. In some example embodiments, thesystems and methods described herein may automatically select a certainregion or part of the image at which to estimate a blur kernel. Such asa region may be selected because the characteristics it may have couldfacilitate accurately and/or effectively deblurring the entire image.The associated blur kernel may then be estimated for the selectedregion.

The systems and methods, in accordance with example embodiments, mayaccess input or reference parameters, such a blur kernel size (e.g., asize automatically determined by the blur kernel sizing engine 150), andidentify a best or suitable region of the image at which to estimateand/or apply the blur kernel. A selected or suitable region may be aregion that includes edges of objects in the image (e.g., an outline ofa person, a tree, etc.), over-exposed pixels within the image,under-exposed pixels within the image, and/or any features within theimage that may break the linearity of motion causing blurring of theimage (e.g., blurring due to camera movement).

An additional input parameter may be associated with a user-identifiedregion within the blurred image, and a best or suitable region that isproximate to the user-defined region may be identified. Accordingly, theuser may provide an initial reference region (or point in a region), andthe user-identified region may be refined or adjusted automatically toenhance deblurring. In some example embodiments, such as when a blurredimage includes a spatially varying blur (e.g., two or more distinctblurred regions within the image), the systems and methods may identifymultiple suitable regions (ideally best regions) in which to estimateand/or apply an associated blur kernel.

FIG. 9 is a block diagram illustrating example components of a regionsuggestion engine 160, in accordance with an example embodiment. Theregion suggestion engine 160 is shown to include a plurality of modulesthat may be implemented in hardware, software, or a combination ofhardware and software, and may be executed by one or more processors.For example, the region suggestion engine 160 may include an imagemodule 910, a blur kernel size module 920, a metrics module 930, and aregion selection module 940. The region suggestion engine 160 mayutilize representative 2D discrete arrays for a selected blurred imagein order to suggest appropriate (ideally best) regions of the images atwhich to estimate and/or apply blur kernels.

A 2D discrete array may often represent a digital image. For example,given a grayscale image f of size W×H, a representative 2D discretearray may be as follows: f(x,y)=f_(x,y), where x∈[0, W−1] and y∈[0, H−1]are integers for pixel coordinates, and f_(x,y) is the intensity valueat the pixel (x,y).

Further, an image gradient ∇f(x,y) that is the first-order discretederivative of f(x,y), may be defined as:∇f(x,y)=(f _(x)(x,y)f _(y)(x,y))^(T)and a discrete partial derivative f_(x)(x,y) may be defined as:f _(x)(x,y)=f(x+1,y)−f(x,y)using the discrete forward difference, and f_(y)(x,y) may be defined ina similar way. Of course, other definitions for the partial derivativesare also possible.

Additionally, as ∇f(x,y) is a 2D vector, its magnitude may be definedas:∥∇f(x,y)∥=√{square root over ({f _(x)(x,y)}² +{f _(y)(x,y)}²)}and its angle may be defined as:

${\phi\left( {\nabla{f\left( {x,y} \right)}} \right)} = {{atan}\left( \frac{f_{y}\left( {x,y} \right)}{f_{x}\left( {x,y} \right)} \right)}$

The above example 2D discrete array may be utilized to represent aselected blurred image in order to suggest suitable regions of theimages at which to estimate and/or apply blur kernels, as described inmore detail below.

Returning to the region suggestion engine 160 shown in FIG. 9, in someexample embodiments, the image module 910 is configured to access ablurred image represented by a 2D discrete array. For example, the imagemodule 910, which may be similar to the image module 210, may access animage selected by a user of the image editing application 135, via theuser interface 115 of the user device 110. It will be appreciated thatimages may be stored in various different locations (e.g., in a photolibrary stored on a device or in the cloud) and such access may bedirectly or via a network (e.g., see FIG. 1).

The blur kernel size module 920 is configured to define a size for ablur kernel associated with the blurred image. In an example embodiment,the blur kernel size module 920 employs similar or the same methodologyas described herein with respect to the blur kernel sizing module 220.Accordingly, the blur kernel size module 920 may include one or moreblur estimators, such as the autocorrelation-based size estimator 222and/or the latent image-based size estimator 224. For example, the blurkernel size module 920 (or, the blur kernel sizing module 220), mayautomatically determine a lower value for a size of a blur kernel forthe blurred region using a first size estimator, automatically determinean upper value for the size of the blur kernel using a different secondsize estimator, and determine a size for the blur kernel that is basedon the lower and upper value for the size of the blur kernel.

In some example embodiments, the image module 910 may access a blurredimage, such as a blurred image b, and preprocess the blurred image bydownsampling the image with respect to the input blur kernel size, suchas a size N_(k)×N_(k). The downsampling may reduce the image size (e.g.,such that computational time is reduced), may reduce the blur, mayreduce noise or artifacts, and so on. The downsampled image b_(low) mayhave a size W_(low)×H_(low).

The metrics module 930 may be configured to determine metrics for atleast two of the plurality of regions based on a number of edgeorientations within a region. The metrics module 930 may perform variousalgorithmic processes to identify image characteristics and/or featuresassociated with favorable or suitable regions within the selected image.A suitable region may be any region determined to be favorable fordeblurring but may not necessarily be the most suitable region withinthe blurred image. Such a region deemed suitable may be based onanalyzing the image to identify contrast edges of many differentdirections, and/or few or no corrupted pixels (e.g., saturated pixels)within the selected image. For example, the metrics module 930 mayidentify regions that include objects having edges of differentorientations to facilitate estimation by the metrics module 930 of ablur kernel. In an example embodiment, a blur kernel may be selected ordefined in a direction orthogonal to an orientation (e.g., edgedirection) of a selected edge. For example, to identify a blur kernel ofsize N_(k)×N_(k), a region may include N_(k) different edge directions.

The metrics module 930 may define an image region R as a set of pixelsin the region, such that:R={(x,y)|R _(left) ≤x<R _(right) ^R _(top) ≤y<R _(bottom)}

-   -   where R_(left) and R_(right) are the leftmost and rightmost x        coordinates, and R_(top) and R_(bottom) are the topmost and        bottommost y coordinates in the region R. The metrics module 930        may then calculate a metric associated with a usefulness U(r) of        a candidate region r, as:        U(R)=(NumEdgeOrientations(R)+α·AvgGradMag(R))·ClipWeight(R)        where NumEdgeDirections(R) is the number of available edge        orientations, AvgGradMag(R) is the average gradient magnitude,        and ClipWeight(R) is a weight for giving penalty to over- or        under-exposed pixels in the region R, and α is a weight for        controlling the amount of contribution of AvgGradMag(R).

In some example embodiments, the variable NumEdgeOrientations(R) is thenumber of available edge orientations in the selected image. To countthe number of orientations, the metrics module 930 may detect edges inthe region R, and uniformly discretize angles into M sections with astep angle θ, such that M·θ=π, where:

$M = {{\max\left( {\left\lceil \frac{N_{k}}{10} \right\rceil,4} \right)}.}$

Each section S_(i) may be defined as S_(i)=[iθ,(i+1)θ), and the metricsmodule 930 may count the number of sections that have sufficient (e.g.,a reference number) of suitable edge pixels, represented as:

${{NumEdgeOrientations}\;(r)} = {\sum\limits_{i = 0}^{M - 1}{H\left\lbrack {{E_{r,i}} - t_{orient}} \right\rbrack}}$

-   -   where H[n] is the Heaviside step function, defined as H[n]=1 if        n≥0, and otherwise H[n]=0. E_(r,i) is the set of edge pixels        whose angles belong to S_(i), defined as E_(r,i)={p|p∈R_(edges)        ϕ(p)∈S_(i)}. |E_(r,i)| is the cardinality or the number of        elements of the set E_(r,i), t_(orient) is a threshold value for        only counting sections having enough edge pixels, R_(edges) is        the set of edge pixels in R detected by edge detection, and ϕ(p)        is the angle of the edge pixel p, where edge pixels are detected        using gradient magnitudes (e.g., when the gradient magnitude of        a pixel is larger than a certain value, the pixel belongs to an        edge).

In some example embodiments, the variable AvgGradMag(R) is the averagegradient magnitude value of the pixels in the region r. For example,when an image region r has more edges of large contrast, thenAvgGradMag(R) becomes large, and vice versa. AvgGradMag(R) may bedefined as:

${{AvgGradMag}\;(R)} = {\frac{1}{R}{\sum\limits_{{({x,y})} \in R}^{\;}{{{\nabla{b_{low}\left( {x,y} \right)}}}.}}}$AvgGradmag(R) is usually a very small number compared toNumEdgeOrientations(R), and may be balanced by a constant α, such asα=10.

In some example embodiments, the variable ClipWeight(R) is defined as:

${{ClipWeight}(R)} = {\sum\limits_{{({x,y})} \in R}^{\;}{c\left( {x,y} \right)}}$

-   -   where c(x,y) is a pixel-wise weight function defined as:

${c\left( {x,y} \right)} = \left\{ \begin{matrix}1 & {{{if}\mspace{11mu} c_{low}} \leq {b_{low}\left( {x,y} \right)} \leq c_{up}} \\{\exp\left( {- \frac{{{{b_{low}\left( {x,y} \right)} - c_{up}}}^{2}}{\sigma_{c}^{2}}} \right)} & {{{if}\mspace{11mu}{b_{low}\left( {x,y} \right)}} > c_{up}} \\{\exp\left( {- \frac{{{{b_{low}\left( {x,y} \right)} - c_{low}}}^{2}}{\sigma_{c}^{2}}} \right)} & {{{if}\mspace{11mu}{b_{low}\left( {x,y} \right)}} < c_{low}}\end{matrix} \right.$

-   -   where c_(up) and c_(low) are thresholds for detecting over- and        under-exposed pixels, respectively, and σ_(c) is a parameter for        controlling the shape of the function c(x,y). For example,

${c_{up} = \frac{220}{255}},{c_{low} = \frac{10}{255}},{{{and}\mspace{14mu}\sigma_{c}} = {\frac{40}{255}.}}$

In some example embodiments, when an image includes spatially varyingblur, blur kernels may be estimated from regions within a center of theimage (or, within a center between the blurred regions). Thesecenter-estimated and/or -applied blur kernels may effectively reduce(ideally remove) the spatially varying blur of the entire image. Thus,in an example embodiment, the metrics module 930 may apply and/or add alocation weight variable to the determined metrics for the regions ofthe image. For example, the metrics module 930 may calculate and/orotherwise determine a usefulness U′(R), which is the usefulness of themetric described above with an additional location weight variable, asfollows:U′(R)=(NumEdgeOrientations(R)+α·AvgGradMag(R))·ClipWeight(R)·LocWeight(R))

-   -   where LocWeight(R) is a weighting function for considering an        image region around an image center (or, another location within        the image), and may be defined as follows:

${{LocWeight}\;(R)} = {\exp\left( {- \frac{{{x_{R} - x_{C}}}^{2} + {{y_{R} - y_{C}}}^{2}}{2\sigma_{L}}} \right)}$

-   -   where (x_(R),y_(R)) is the center pixel of the image region R,        and (x_(C),y_(C)) is the image center, and        σ_(L)=max(W_(low),H_(low))/2.

In some example embodiments, the metrics module 930 may receiveinformation (e.g., a user input from a mouse, touchpad, touch screen, orthe like) that identifies a user-defined location and/or region withinthe image at which to estimate a blur kernel. The metrics module 930 maydefine and/or modify the weighting function LocWeight(R) as follows:

${{LocWeight}\;(R)} = {\exp\left( {- \frac{{{x_{R} - x_{i}}}^{2} + {{y_{R} - y_{i}}}^{2}}{2\sigma_{L}}} \right)}$

-   -   where (x_(i),y_(i)) is the pixel of the interest that user        provided, and σ_(L)=max(W_(low),H_(low))/4. Accordingly, in an        example embodiment, a user may at least partially determine a        location at which a blur kernel is to be positioned.

In some example embodiments, the metrics module 930 may determinemetrics for regions of varying sizes. For example, small regions (e.g.,regions less than a reference size) of an image may include enoughcharacteristics (e.g., edge orientations) to suitably perform blurkernel estimation. The metrics module 930 may then determine metrics forthe small regions in order to reduce computation times, avoid regionswith corrupted pixels, and so on. Of course, the metrics module 930 maydetermine metrics for regions within an image based on other processes,techniques, and/or algorithms that include variables associated withedges, edge orientations, and so on, within the regions.

In some example embodiments, the region selection module 940 isconfigured to select a region based on the determined metrics at whichto position the blur kernel during deconvolution of the blurred image.For example, the region selection module 940 may select the region orregions associated with the highest values of the usefulness U(R) and/orU′(R).

In order to identify a suitable region, the region selection module 940may consider various region sizes (e.g., using a pre-defined set ofscale factors) along with the metrics associated with the region sizes.When a suitable region is identified, the region selection module 940may select the region with the determined scale factor size and ignoreother sizes. However, if a suitable region, or a region not meetingdefined criteria, is not found, then the region selection module 940multiplies a next scale factor with the blur kernel size, and selects aregion of that size, until a suitable or best region is identified.

For example, the region selection module 940 may locate a region with alarge or largest usefulness U(R) or U′(R), and determine whether theusefulness U(R) or U′(R) is greater than a threshold value, t_(u) Thethreshold value t_(u) may define a minimum or sufficient value for thesuitability of a region at which to estimate and/or apply a blur kernel.Thus, once the metrics module 930 calculates metrics for some or allregions of an image, the region selection module 940 may select theregion or regions having the highest usefulness U(R) or U′(R) and/or anyregions with a threshold value t, above a reference threshold t_(u),t_(u)=max(0.75M, 4), which indicates a region is suitable for blurkernel estimation.

FIG. 10 shows a screen shot 1000 of suggested regions for automaticallyestimating a blur kernel. More particularly, the screen shot 1000 showsa first region 1010 and a second region 1015 at which to estimate a blurkernel. A blur kernel 1020 may be estimated for the first region 1010,and a blur kernel 1025 may be estimated for the second region 1015. Themetrics module 930 may then, for example, determine which of the firstand the second regions 1010, 1015 is more suitable for estimating a blurkernel used to deblur the image. For example, the regions may beselected based on a number of edge orientations within each region

In an example embodiment, when the determined metrics do not satisfy athreshold metric for selecting the region, the size of the regions maybe modified. For example, the size of each of a plurality of regions maybe modified, and metrics for each the plurality of regions having amodified size may then be determined. Thereafter, a region having themodified size that is associated with a metric that satisfies thethreshold metric for estimating the blur kernel may be selected.

As described herein, in an example embodiment, the region suggestionengine 160 may perform various techniques to automatically select and/ordetermine a region within an image at which to estimate a blur kerneland/or apply the blur kernel to deblur the image. FIG. 11 is a flowdiagram illustrating a method 1100, in accordance with an exampleembodiment, for selecting a region in an image for estimating a blurkernel. The method 1100 may be performed by the region suggestion engine160 and, accordingly, is described herein merely by way of example withreference thereto. It will be appreciated that the method 1100 may beperformed on any suitable hardware.

In operation 1110, the region suggestion engine 160 accesses a blurredimage having a plurality of regions. One or more of the regions may beblurred regions. In an example embodiment, the blurred image may beaccessed by the image editing application 135, via the user interface115 of the user device 110.

In operation 1120, the region suggestion engine 160 defines a size foreach of the plurality of regions. In some example embodiments, theregion suggestion engine 160 may preprocess the blurred image bydownsampling the image with respect to the input blur kernel size, suchas a size N_(k)×N_(k). The downsampling may reduce the image size andoutput a downsampled image b_(low), having a size W_(low)×H_(low). Thedownsampling may reduce computational time, may reduce the blur, mayreduce noise or artifacts, and so on.

In operation 1130, the region suggestion engine 160 determines metricsfor at least two of the plurality of regions, the metrics being based ona number of edge orientations within each region. For example, themetrics module 930 may perform various algorithmic processes describedherein to identify image characteristics and/or features associated withsuitable regions within the accessed image. In an example embodiment,the suitable regions may have contrast edges of many different edgedirections, few or no corrupted pixels (e.g., saturated pixels), and soon. Example metrics may be based on edge orientations, a gradientmagnitude, a clip weight, a location weight, and combinations thereof.

Thereafter, as shown, in operation 1140, the region suggestion engine160 selects a region, based on the determined metrics, at which toestimate and/or position the blur kernel during deconvolution of theblurred image. For example, the region selection module 940 may selectthe region or regions associated with the highest usefulness metric U(R)and/or U′(R) values. FIG. 12 is an example of a blurred image 1200showing positioning of an example blur kernel 1220 at a selected region1210 of the blurred image 1200.

The method 1100 may then, as shown in operation 1150, estimate a blurkernel for deblurring the image based on the selected region.Thereafter, as shown in operation 1160, the image may be deblurred usingthe blur kernel.

As described herein, in some example embodiments, the region suggestionengine 160 may identify multiple suitable regions in the blurred image1200 for blur kernel estimation. To this end, in an example embodimentthe region selection module 940 may determine whether any of theidentified regions overlap and, if the regions overlap by more than acertain ratio or reference amount, other suitable regions may then beselected for blur kernel estimation.

In some example embodiments, the region suggestion engine 160 maydetermine a metric for a single region within the image (e.g., auser-defined or center region), and, if the metric indicates the regionis suitable for blur kernel estimation, select the single region withoutcalculating metrics for other regions of the image.

Thus, the region suggestion engine 160 may select a region within animage at which to estimate and/or position a blur kernel, in order todeblur the image.

Examples of Deblurring Images Having Spatially Varying Blur

Taking handheld photos (e.g., using a smartphone, a mobile device, or acamera) in low-light conditions may be challenging, as longer exposuretimes are needed, often leading to blurred photos due to the shaking ofthe camera. Photos taken of scenes having depth variation may havedifferent spatially varying blur, because objects far away from thecamera will be less blurred than objects closer to the camera.Accordingly, in an example embodiment, the image editing application 135may utilize multiple blur kernels to deblur the photos. For example, ablur kernel may be provided for each different blurred region within animage. Because differences between blur kernels for an image withspatially varying blur may be small, the image editing application 135may deblur a single, local, region within an image using an associatedstatic blur kernel. Different blur kernels may then be used in otherregions and the deblurred results may be combined or stitched togetherto form a final deblurred image (e.g., see FIG. 15).

FIG. 13 is a block diagram of example components of an image deblurringengine 140, in accordance with an example embodiment, to reduce (ideallyremove) spatially varying blur using multiple blur kernels. In someexample embodiments, the image deblurring engine 140 may include one ormore modules implemented in hardware, software, or a combination ofhardware and software, and that may be executed by one or moreprocessors. The image deblurring engine 140 is shown, by way of example,to include an image module 1310, a blur kernel module 1320, a kernelpositioning module 1330, and a deblurring module 1340.

In some example embodiments, the image module 1310 is configured toaccess an image having multiple blurred regions (e.g., two or more),such as a first blurred region and a second blurred region. For example,the image module 1310, which may be similar to the image module 210and/or image module 910, may access a blurred image resident on a localdevice, in the cloud, or otherwise. In an example embodiment, the imagemodule 1310 accesses an image input to the image editing application135, via the user interface 115 of the user device 110.

In some example embodiments, the blur kernel module 1320 is configured(e.g., programmed) to generate a first blur kernel for the first blurredregion and a second blur kernel for the second blurred region. Forexample, the blur kernel module 1320 may generate and/or determine localblur kernels for some or all of the blurred regions within the imageusing techniques described herein.

In some example embodiments, the image editing application 135 mayprovide a graphical user interface (GUI), such as one or more of thevarious user interfaces described herein, configured to receive inputfrom a user that specifies local regions within the image for blurkernel estimation. For example, in response to a user providing input(e.g., drawing a box on the image via the GUI, centered at (x,y)), theblur kernel module 1320 may analyze the image content within theidentified region to estimate a static or local blur kernel K for theregion. The blur kernel module 1320, which may use various techniquesfor generating blur kernels, such as those described herein, may thencreate a list of local kernels K₁, K₂, . . . , K_(n), which are createdfrom regions centered at (x₁,y₁), (x₂,y₂), . . . , (x_(n),y_(n)). Inother example embodiments, the image editing application 135 may performthe techniques described herein to automatically suggest the localregions for kernel estimation. Thus, the blur kernel module 1320 maygenerate a group of local blur kernels that are used to at leastpartially (ideally satisfactorily) deblur a local region. Thus, a singleblur kernel is not used to deblur all blurred regions of entire image.

In some example embodiments, the kernel positioning module 1330 isconfigured to position the first blur kernel with respect to the firstblurred region and position the second blur kernel with respect to thesecond blurred region based on the position of the first blur kernel.For example, the kernel positioning module 1330 may align the local blurkernels with respect to each other.

Given a list of generated blur kernels, K₁, K₂, . . . , K_(n), whichcorrespond to local blurred regions within the image, the kernelpositioning module 1330 may identify a blur kernel K_(c) having a centerposition (x_(c),y_(c)) that is closest to a center of the image orclosest to a focal point of the image (e.g., a focal point identifiedfrom metadata associated with the image). Thereafter, the remaining blurkernels are positioned relative to the blur kernel K_(c), which maydefine a center kernel or base kernel.

For example, for a local blur kernel K_(r) that is created from a regioncentered at (x_(r),y_(r)), the kernel positioning module 1330 mayidentify an optimal shift (δx*,δy*), as follows:

${\left( {{\delta\; x^{*}},{\delta\; y^{*}}} \right) = {\arg\;{\max\limits_{{\delta\; x},{\delta\; y}}{\sum\limits_{x,y}^{\;}{{K_{c}\left( {x,y} \right)} \cdot {K_{r}\left( {{x + {\delta\; x}},{y + {\delta\; y}}} \right)}}}}}},$

-   -   where x, y iterate over all pixels in the kernel. In example        embodiment, the kernel positioning module 1330 may thus ideally        determine an optimal shift between the blur kernel (x_(r),y_(r))        and the base kernel K_(c) that results in an enhanced (ideally a        maximum or high) correlation between the base kernel K_(c) and        the local blur kernel K_(r).

In some example embodiments, the deblurring module 1340 is configured(e.g., programmed) to deconvolve the first blurred region with the firstblur kernel and deconvolve the second blurred region with the secondblur kernel. For example, the deblurring module 1340 may performtile-based multi-kernel deconvolution to deblur the entire image usingthe aligned, local, static kernels. Tile-based multi-kerneldeconvolution may comprise dividing up a blurred image into a pluralityof regions or tiles that do not overlap. In an example embodiment, thetiles may be rectangular or square tiles that are arranged in a grid.For example, the tile size can be a fixed number, such as 512 by 512pixels.

In an example embodiment, the deblurring module 1340 uses each localkernel K_(i) to generate a deconvolution result L_(i) for each localregion (e.g., tile) of the image, and may blend the deconvolutionresults (L_(i)s) together, using various techniques, to form a final,deblurred image.

FIG. 14 is a flow diagram illustrating a method 1400, in accordance withan example embodiment, for deblurring a blurred image. The method 1400may be performed by the image deblurring engine 140 and, accordingly, isdescribed herein merely by way of example with reference thereto.However, it will be appreciated that the method 1400 may be performed onany suitable hardware.

The method 1400 is shown to commence at operation 1410 wherein the imagedeblurring engine 140 accesses an image having a first blurred region(e.g., centered at (x₁,y₁)) and a second blurred region (e.g., centeredat (x₂,y₂)). Thus, the image module 1310 may access a blurred imagehaving spatially varying blur, such as two or more blurred regions thatare spatially offset.

Thereafter, in operation 1420, a first blur kernel (e.g., K₁) isgenerated for the first blurred region and a second blur kernel (e.g.,K₂) is generated for the second blurred region. When multiple blurredregions are present, the blur kernel module 1320 may generate furtherlocal blur kernels for each of the blurred regions within the image. Thefirst blur kernel is positioned with respect to the first blurredregion, and the second blur kernel is positioned with respect to thesecond blurred region based on the position of the first blur kernel(see operations 1430 and 1440). For example, the kernel positioningmodule 1330 may consider the first blur kernel as a base or centerkernel, and locate, position, and/or align the second blur kernel withrespect to non-zero elements of the first blur kernel such that anoptimal shift between the second blur kernel and the base kernel ismaximized or highly correlated.

FIG. 15 is an example blurred image 1500 showing positioning of multipleblur kernels, in accordance with an example embodiment, at multipleblurred regions of the image 1500. As shown by way of example, a blurkernel 1515 is positioned proximate to a region 1510, a blur kernel 1525is positioned proximate to a region 1520, a blur kernel 1535 ispositioned proximate to a region 1530, and a blur kernel 1545 ispositioned proximate to a region 1540. Although the method 1400 isdescribed with reference to two blur kernels, it is to be appreciatedthat multiple blur kernels may be utilized (e.g., four blur kernels areshown by way of example in FIG. 15).

As shown in operation 1450, the image may be deblurred by deconvolvingthe first blurred region (e.g., region 1510) with the first blur kernel(e.g., blur kernel 1515) and deconvolving the second blurred region(e.g., region 1520) with the second blur kernel (e.g., blur kernel1525). For example, the deblurring module 1340 may generate adeconvolution result for the first blurred region and generate adeconvolution result for the second blurred region, and blend theresults to create a final, deblurred image. In a similar fashion,deconvolution of further blur kernels and further blurred regions maytake place (e.g., regions 1530 and 1540 with blur kernels 1535 and1545). Deconvolution, as described herein, is intended to include areverse operation of a convolution operation, and, therefore, mayreverse, change, and/or modify blur, distortion, or other imperfectionsin images.

In the example blurred image 1500, blurred regions 1510 and 1530 areshown by way of example to overlap in region 1528. In such circumstanceswhere two or more blurred regions overlap, the deblurring module 1340may perform a variety of different techniques when blendingdeconvolution results to create a final, deblurred image. For example,FIG. 16A shows a diagram 1600 that displays a blending region 1615between a first region 1610 occupied by a first blur kernel (e.g., blurkernel K1) and a second region 1620 occupied by a second blur kernel(blur kernel K2), wherein the first and second regions 1610, 1620overlap. The first region 1610 is shown to be centered at (x₁,y₁) in theimage and the second region 1620 is centered at (x₂,y₂) in the image.For example, using the methodology described herein, deconvolutionresults L₁ and L₂ may be determined using blur kernels K₁ and K₂, havingcenters at pixels (x₁, y₁) and (x₂,y₂), respectively, in the blurredimage.

For any given pixel (x,y), in an overlapping or blending region 1615,the deblurring module 1340 may assign two different weights, w₁ and w₂,and determine a color for the pixel (x,y) as follows:C(x,y)=w ₁ L ₁(x,y)+w ₂ L ₂(x,y),which is a linear interpolation between the two resultant images L₁ andL₂ that include the pixel.

To determine the weights, a spatial distance between the pixel and thetwo kernel centers ((x₁,y₁), (x₂,y₂)) may be calculated. The spatialdistance may be denoted by d_(a1) and d_(a2), and the weights may becomputed as follows:

$\begin{matrix}{{w_{1} = 1},{w_{2} = 0}} & {{{{if}\mspace{11mu} d_{1}} + D} < d_{2}} \\{{w_{1} = 0},{w_{2} = 1}} & {{{{if}\mspace{11mu} d_{2}} + D} < d_{1}}\end{matrix}$${w_{1} = {0.5 - \frac{d_{1} - d_{2}}{2D}}},{w_{2} = {1 - w_{1}}},$esle

-   -   where D is a distance threshold, (e.g., 20 pixels). Thus, when        the given pixel (x,y) is close to the center of kernel K1 in the        first region 1610, but is far away (e.g., greater than a        reference distance) from the center of kernel K2 in the second        region 1620, then w₁=1 and w₂=0, the pixel will have a color        similar to the color of L₁. On the other hand, when the pixel is        far away from the center of kernel K1, and is close to the        center of kernel K2, the pixel may have a color similar to the        color of L₂. However, when the pixel is located within the        blending region 1615, having a width D and located between the        two kernels K1, K2, the pixel at (x,y) will have a color that is        a linear blend of the corresponding colors from L₁ and L₂. Thus,        the deblurring module 1340 may linearly blend resultant image        portions L₁ and L₂ and perform a final deconvolution result for        the entire image. It should be noted that, in other embodiments,        other interpolation techniques are utilized.

In some example embodiments where a pixel is common to more than twokernels, blending weights may be determined for all kernels that includethe pixel. For example, given m kernels K₁, K₂, . . . , K_(m), weightsbetween any pair of kernels as w_(ij), i, j∈{1, . . . , m} may bedetermined. For example, the deblurring module 1340 may determine anun-normalized weight w_(i) for Kernel i as follows:w _(i) =w _(i1) ·w _(i2) · . . . ·w _(im).

The deblurring module 1340 may then normalize the weights using the sumof the un-normalized weights as:

$w_{i}^{\prime} = {\frac{w_{i}}{\sum\limits_{i = 1}^{m}w_{i}}.}$and utilize the weights to linearly blend the pixel colors of all L_(i)sfor the pixel. This weighting functionality may, for example, be appliedto the example kernels 1515, 1525, 1535 and 1545 of the blurred image1500 (see FIG. 15).

In some example embodiments, the deblurring module 1340 may utilize anefficient, tile-based blending technique, where tiles (e.g., groups ofpixels) are blended between kernels. An example of a tile-based blendingis shown in FIG. 16B. More particularly, FIG. 16B shows a diagram 1650that displays a blending region 1675 between a first region 1660occupied by a first blur kernel K1 and a second region 1670 occupied bya second kernel K2. An image tile (e.g., Tile 1 or Tile 2) may bedefined having four corner locations at P1, P2, P3 and P4. Blendingweights may then be determined for the four corner locations P1, P2, P3and P4. When the four corner locations P1, P2, P3 and P4 are located ina single (the same) kernel (e.g., have a weighted value of 1 for thesame kernel, and have a weighted value 0 for all other kernels), it isassumed that the tile is located entirely within a single kernel region(e.g., Tile 1), and the single kernel is then used to deconvolve theentire tile.

However, when the four locations P1, P2, P3 and P4 of the corers havedifferent weights for different kernels, the tile may then be assumed tobe located within two or more kernel regions (e.g., Tile 2). The weightsof the four corner locations P1, P2, P3 and P4 may be examined in orderto identify all kernels that have non-zero weights with the tile,determine the corresponding blending results for the kernels, andlinearly blend the kernels in order to deblur the entire image withsmooth transitions in color between different regions of the image.

Thereafter, the deblurring module 1340 may stitch regions together,using various techniques to form complete deblurred images. Stitchingregions together may include computing multiple colors for a pixel, eachassociated with an estimated blur kernel, and finally computing aweighted average of these colors as its final color.

Thus, in some example embodiments, the image deblurring engine 140 maydeblur an image having spatially varying blur using multiple kernels andwherein the kernels may be aligned with respect to one another. Blendingregions within the image that include pixels associated with multiplekernels may be deconvolved using weights that are dependent upon pixeldistances from local kernels.

Examples of Graphical User Interfaces for Managing Blur Kernels

In an example embodiment, a display interface and a method forpresenting editing controls for deblurring an image are provided. Thedisplay interface and the method may form part of the image editingapplication 135 and, various user interface screens may be provided thatfacilitate the management of blur kernels used to deblur images.

Referring to FIG. 17, a graphical user interface (GUI) 1700, inaccordance with an example embodiment, is provided for editing images(e.g., photos in a photo library). The GUI 1700 may be displayed on amobile device (e.g., a tablet computer, smartphone or the like) or anyother computing device. The GUI 1700 is shown to include a display zone1710 that displays an image 1715, such as a blurred image, and a controlzone 1720 that includes various user input controls or features that areused to deblur and/or otherwise edit the image 1715. For example, thecontrol zone 1720 is shown to include various blur trace settings 1725,a blur kernel zone 1727, and an image detail zone 1729. The blur kernelzone 1727 is shown by way of example to include a blur trace poundsslider, which may specify a size of the blur kernel (e.g., if the slideris set to 41, then the size of the blur kernel is set to 41×41 pixels),a smoothing slider, which may specify how smooth the deblurred resultwill be, or how much noise is suppressed in the deblurred result, and anartifact suppression slider, which may specify how much of thedeblurring artifacts (e.g., ringing artifacts, will be suppressed). Inother example embodiments, different and/or additional settings areprovided using sliders and/or other adjustment arrangements. The blurkernel zone 1727 is shown, by way of example, to display threevisualizations of blur kernels. More particularly, blur kernels 1730,1732 and 1734 may be applied to the image 1715 to deblur blurred regionsof the image 1715. In an example embodiment, the blur kernels 1730, 1732and 1734 are generated using the methods described herein. The imagedetail zone 1729 shows an exploded view of a region 1716 of the image1715.

In some example embodiments, parameters or aspects (e.g., size) of theblur kernels 1730, 1732 and 1734, that are displayed within the blurkernel zone 1727, are automatically configured or defined without userinput. Further, regions in the image 1715 (e.g., the blurred region1716) at which to apply blur kernels (e.g., the blur kernels 1730, 1732and 1734) may be automatically determined and identified using thetechniques described herein. Examples of components of a displayinterface that generates the GUI 1700 are shown in FIG. 18.

More particularly, a display interface 1800, in accordance with anexample embodiment, is shown by way of example to include an accessmodule 1810 and a display module 1820. The access module 1810 isconfigured (e.g., by a processor executing instructions) to access ablurred image (e.g., access image data defining pixels of an image), andthe display module 1820 is configured by at least one processor todisplay a graphical user interface for deblurring regions of the image.In an example embodiment, the display interface 1800 may generate theGUI 1700 and, accordingly, the display module 1820 may be configured tocause the display of the display zone 1710 and the control zone 1720adjacent to the display zone 1710. It should be noted that the positionof the control zone 1720 relative to the display zone 1710 may vary fromone embodiment to another. Thus, the control zone 1720 may be below, ontop of, or otherwise positioned relative to the display zone 1710. Thedisplay module 1820 is configured to display a user selected image(e.g., the blurred image 1715), display one or more suggested blurkernels (e.g., the blur kernels 1730, 1732 and 1734) in the control zone1720, and display an association between a selected blur kernel and ablurred region (e.g., the blurred region 1716). The access module 1810and the display module 1820 may be implemented in hardware, software, ora combination of hardware and software, and may be executed by one ormore processors. In some example embodiments, the access module 1810 isconfigured to access a blurred image stored locally and/or remotely. Forexample, the access module 1810 may be similar to the image module 210,the image module 910, and/or image module 1310.

The display module 1820 may cause the blurred image 1715 to be displayedwithin the display zone 1710, as well as an indication of anautomatically suggested region (e.g., the blurred region 1716) at whichto estimate and/or apply a blur kernel. The display module 1820 may alsobe configured to cause the display of user input controls to allow auser to edit and deblur images. For example, the display module 1820 maycause various editing buttons, sliders or the like to be displayedwithin the control zone 1720 that are used to edit the image 1715. In anexample embodiment, proposed blur kernels are automatically generatedand displayed in the blur kernel zone 1727, and the display module 1820monitors user interaction with the GUI 1700 and deblurs the blurredimage 1715 responsive to the user interaction. In an example embodiment,the display module 1820 may render and/or cause blur kernels to bedisplayed within the blur kernel zone 1727, and/or cause the blur tracesettings 1725 to be displayed within the control zone 1720.

FIG. 19 is a flow diagram of a method 1900, in accordance with anexample embodiment, for presenting editing controls for deblurring animage. The method 1900 may be performed by the image editing application135 and/or the display interface 1800 and, accordingly, is describedherein merely by way of reference thereto. It will be appreciated thatthe method 1900 may be performed on any suitable hardware.

The method 1900, in operation 1910, causes display of a graphical userinterface configured to be used to deblur an image. The graphical userinterface may include a display zone and a control zone adjacent to thedisplay zone. A user selected image is displayed in the display zone asshown in operation 1920. Thereafter, a suggested blur kernel isdisplayed in the control zone (see operation 1930), the blur kernelbeing associated with a blurred region in the selected image. In anexample embodiment, in addition to displaying the blur kernel (or aplurality of blur kernels) in the control zone, the suggested blurkernel is also displayed proximate the associated blurred region in thedisplay zone (see operation 1832).

Displaying the suggested blur kernel proximate the associated region maycomprise overlaying the blur kernel on the blurred region. Further, themethod 1900 may display a plurality of blur kernels in the control zone(e.g., see FIG. 17) wherein each of the plurality of blur kernels isassociated with a corresponding blurred region, as described in furtherdetail below. As described herein, in some example embodiments, theimage editing application 135, via the GUI 1700, provides various userinput controls (e.g., controls of control zone 1720) that facilitatemanagement of blur kernels.

In some example embodiments, the GUI 1700 may provide visual elementsand/or objects associated with some or all of the blur kernels to beutilized in deblurring an image. For example, the blur kernel zone 1727may provide a kernel thumbnail list in which all kernels are representedby “thumbnails” of kernel images. The blur kernel zone 1727 may providevisualizations for all blur kernels as well as their relationship withthe image 1715. FIGS. 20A-20B are exploded views of the blur kernel zone1727 of the GUI 1700 shown FIG. 17. The blur kernel zone 1727 is shownto include three blur kernels but it is to be appreciated that fewer ormore blur kernels may be present. Further, in order to select ordeselect a particular blur kernel 1730, 1732 or 1734, checkboxes 1742,1744 and 1746 are provided. Hence, using the GUI 1700, a user may managewhich blur kernels are used to deblur various blurred regions in animage. In order to allow a user to add further blur kernels, an addfeature in the form of a “+” button 1748 is provided. When the userclicks on the “+” button a further blur kernel is added to the blurkernel zone 1727. Likewise, the GUI 1700 may include a feature (e.g., abutton) to remove blur kernels from the blur kernel zone 1727, Theparameters (e.g., size, location, etc) may be automatically determinedusing the methodologies described herein. When a checkbox 1740 isselected, then regions in the image (e.g., the blurred region 1716) thatcorrespond to a blur kernel are shown. In some example embodiments, asshown in FIG. 20B, when a user selects or otherwise identifies (e.g., bya mouse cursor hovering over) a blur kernel, the blur kernel isenlarged. In FIG. 20B blur kernel 1732 is shown to be enlarged.

In an example embodiment, drag-and-drop functionality is provided thatallows a user to drag a blur kernel (e.g., one or more of the blurkernels 1730, 1732 and 1734) and drop it at a position or location inthe blurred image 1715. Deblurring of the image may then beautomatically performed. In an example embodiment, a user may identify ablurred region in the display zone 1710 (e.g., the blurred region 1716)using a pointer of a mouse, a finger on a touch screen, or the like),and suggested blur kernels for the selected blurred region may then beautomatically determined and displayed in the blur kernel zone 1727.Estimation of the size and other parameters of the blur kernel may bedone using any of the methodologies described herein. The GUI 1700 mayprovide controls to create, edit, and/or delete blur kernels (e.g., seebuttons 1702 and 1704 in FIG. 17). In an example embodiment, the usermay draw a rectangle around a blurred region of the image to select theblurred region. Thus, in an example embodiment, a blur boundaryindicator and a kernel boundary indicator may be displayed on theselected image. The blur boundary indicator may identify a size of theblurred region to be processed, and the kernel boundary indicator mayidentify a size of the associated blur kernel. Accordingly, in anexample embodiment, the blur boundary indicator and the kernel boundaryindicator may show the relative sizes of the blurred region and the blurkernel.

In some example embodiments, the image editing application 135 mayautomatically suggest the region and create the blur kernel for theregion, as described herein. For example, the image editing application135 may identify a preferable (ideally best) region within the image atwhich to estimate the blur kernel. Accordingly, the functionalitydescribed in the method 300, wherein a suggested size for a blur kernelis determined, may be performed using the example GUI 1700. The GUI 1700may also be used in performing the other methodologies described herein.

In some example embodiments, the GUI 1700 may provide interactiveelements and/or indicators that reveal regions within an image that areassociated with a blur kernel. Example indicators may include indicatorsthat identify whether a kernel is activated (e.g., a box is checked nextto the kernel or a circle within a center of a box identifying a regionis highlighted), a blur selection ring around a blur kernel isdisplayed, the blur kernel and/or the associated region is highlightedin response to a mouse hovering over the region or the blur kernel, andso on.

In some example embodiments, the GUI 1700 may provide controls used tozoom in or out of blur kernels at various levels of granularity. FIG.20C shows an exploded view of the kernel zone 1727 where a user hasactivated an option menu 1750 (e.g., using a right-click function on amouse or any gesture). The example option menu 1750 is shown to includefunctions including “Start Estimation,” “Scroll to Estimation Region,”“Show Blur Trace on Image,” “Save Blur Trace . . . ” and “Delete BlurTrace.” For example, when the “Show Blur Trace on Image” option isselected, a corresponding blur kernel image may be overlaid on theimage. Various other functions associated with deblurring an image maybe provided in other example embodiments.

As mentioned above, a plurality of blur kernels may be displayed in thecontrol zone, each of the plurality of blur kernels being associatedwith a corresponding blurred region. Each of the plurality of blurkernels may then be displayed proximate its associated blurred region inthe display zone.

In an example embodiment, the display zone (e.g., the display zone 1710)defines a preview canvas for displaying changes to a selected image(e.g., the blurred image 1715) responsive to changes in one or more blurkernel parameters (e.g., using functionality provided in the controlzone 1720). The GUI 1700 may also provide functionality to allow a userto define or input kernel parameters. These parameters may allow a userto modify at least one parameter of the blur kernel. An example of suchan input is the size of the blur kernel, and the image in the displayzone may then be modified in response to a variation in the at least oneparameter.

As mentioned above, displaying the suggested blur kernel proximate theassociated region may comprise overlaying a blur kernel on the blurredregion (e.g., the blurred region 1716 of the blurred image 1715). FIG.21 is a GUI 2100, in accordance with an example embodiment, illustratingan overlay of a blur kernel 2110 on an associated image. The GUI 2100may display the blur kernel 2110 within the display zone with the samezoom level of the display zone 1710, which provides information as tothe relative size between the blur kernel 2110 and the blurred region(e.g., the blurred region 1716) in the blurred image 1715. A user may beable to zoom in or out on the blur kernel 2110 at arbitrary levels.Also, the blur kernel 2110 or blur trace is generally similar to theblur within the image 1715, and displaying the blur kernel 2110proximate to the image 1715 may enable the user to review the quality ofthe kernel estimation with respect to the image 1715.

FIG. 22 is a block diagram illustrating components of a machine 2200,according to some example embodiments, able to read instructions from amachine-readable medium (e.g., a machine-readable storage medium) andperform any one or more of the methodologies discussed herein, in wholeor in part. Specifically, FIG. 22 shows a diagrammatic representation ofthe machine 2200 in the example form of a computer system and withinwhich instructions 2224 (e.g., software, a program, an application, anapplet, an app, or other executable code) for causing the machine 2200to perform any one or more of the methodologies discussed herein may beexecuted. In alternative embodiments, the machine 2200 operates as astand-alone device or may be connected (e.g., networked) to othermachines. In a networked deployment, the machine 2200 may operate in thecapacity of a server machine or a client machine in a server-clientnetwork environment, or as a peer machine in a peer-to-peer (ordistributed) network environment. The machine 2200 may be a servercomputer, a client computer, a personal computer (PC), a tabletcomputer, a laptop computer, a netbook, a set-top box (STB), a personaldigital assistant (PDA), a cellular telephone, a smartphone, a webappliance, a network router, a network switch, a network bridge, or anymachine capable of executing the instructions 2224, sequentially orotherwise, that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include a collection of machines that individually orjointly execute the instructions 2224 to perform any one or more of themethodologies discussed herein.

The machine 2200 includes a processor 2202 (e.g., a central processingunit (CPU), a graphics processing unit (GPU), a digital signal processor(DSP), an application specific integrated circuit (ASIC), aradio-frequency integrated circuit (RFIC), or any suitable combinationthereof), a main memory 2204, and a static memory 2206, which areconfigured to communicate with each other via a bus 2208. The machine2200 may further include a graphics display 2210 (e.g., a plasma displaypanel (PDP), a light emitting diode (LED) display, a liquid crystaldisplay (LCD), a projector, or a cathode ray tube (CRT)). The machine2200 may also include an alphanumeric input device 2212 (e.g., akeyboard), a cursor control device 2214 (e.g., a mouse, a touchpad, atrackball, a joystick, a motion sensor, or other pointing instrument), astorage unit 2216, a signal generation device 2218 (e.g., a speaker),and a network interface device 2220.

The storage unit 2216 includes a machine-readable medium 2222 on whichis stored the instructions 2224 embodying any one or more of themethodologies or functions described herein. The instructions 2224 mayalso reside, completely or at least partially, within the main memory2204, within the processor 2202 (e.g., within the processor's cachememory), or both, during execution thereof by the machine 2200.Accordingly, the main memory 2204 and the processor 2202 may beconsidered as machine-readable media. The instructions 2224 may betransmitted or received over a network 2226 (e.g., network 120 ofFIG. 1) via the network interface device 2220.

As used herein, the term “memory” refers to a machine-readable mediumable to store data temporarily or permanently and may be taken toinclude, but not be limited to, random-access memory (RAM), read-onlymemory (ROM), buffer memory, flash memory, and cache memory. While themachine-readable medium 2222 or computer-readable medium is shown in anexample embodiment to be a single medium, the term “machine-readablemedium” or “computer-readable medium” should be taken to include asingle medium or multiple media (e.g., a centralized or distributeddatabase, or associated caches and servers) able to store instructions2224. The term “machine-readable medium” or “computer-readable medium”shall also be taken to include any medium, or combination of multiplemedia, that is capable of storing instructions (e.g., instructions 2224)for execution by a machine or computer (e.g., machine 2200), such thatthe instructions, when executed by one or more processors of the machineor computer (e.g., processor 2202), cause the machine or computer toperform any one or more of the methodologies described herein.Accordingly, a “machine-readable medium” refers to a single storageapparatus or device, as well as “cloud-based” storage systems or storagenetworks that include multiple storage apparatuses or devices. The term“machine-readable medium” shall accordingly be taken to include, but notbe limited to, one or more data repositories in the form of asolid-state memory, an optical medium, a magnetic medium, or anysuitable combination thereof.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute eithersoftware modules (e.g., code embodied on a machine-readable medium or ina transmission signal) or hardware modules. A “hardware module” is atangible unit capable of performing certain operations and may beconfigured or arranged in a certain physical manner. In various exampleembodiments, one or more computer systems (e.g., a standalone computersystem, a client computer system or a server computer system) or one ormore hardware modules of a computer system (e.g., a processor or a groupof processors) may be configured by software (e.g., an application orapplication portion) as a hardware module that operates to performcertain operations as described herein.

In some example embodiments, a hardware module may be implementedmechanically, electronically, or any suitable combination thereof. Forexample, a hardware module may include dedicated circuitry or logic thatis permanently configured to perform certain operations. For example, ahardware module may be a special-purpose processor, such as a fieldprogrammable gate array (FPGA) or an ASIC. A hardware module may alsoinclude programmable logic or circuitry that is temporarily configuredby software to perform certain operations. For example, a hardwaremodule may include software encompassed within a general-purposeprocessor or other programmable processor. It will be appreciated thatthe decision to implement a hardware module mechanically, in dedicatedand permanently configured circuitry, or in temporarily configuredcircuitry (e.g., configured by software) may be driven by cost and timeconsiderations.

Accordingly, the phrase “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. As used herein,“hardware-implemented module” refers to a hardware module. Consideringembodiments in which hardware modules are temporarily configured (e.g.,programmed), each of the hardware modules need not be configured orinstantiated at any one instance in time. For example, where a hardwaremodule comprises a general-purpose processor configured by software tobecome a special-purpose processor, the general-purpose processor may beconfigured as respectively different special-purpose processors (e.g.,comprising different hardware modules) at different times. Software mayaccordingly configure a processor, for example, to constitute aparticular hardware module at one instance of time and to constitute adifferent hardware module at a different instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multiplehardware modules exist contemporaneously, communications may be achievedthrough signal transmission (e.g., over appropriate circuits and buses)between or among two or more of the hardware modules. In embodiments inwhich multiple hardware modules are configured or instantiated atdifferent times, communications between such hardware modules may beachieved, for example, through the storage and retrieval of informationin memory structures to which the multiple hardware modules have access.For example, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation),

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions describedherein. As used herein, “processor-implemented module” refers to ahardware module implemented using one or more processors.

Similarly, the methods described herein may be at least partiallyprocessor-implemented, with a processor being an example of hardware.For example, at least some of the operations of a method may beperformed by one or more processors or processor-implemented modules.Moreover, the one or more processors may also operate to supportperformance of the relevant operations in a “cloud computing”environment or as a “software as a service” (SaaS). For example, atleast some of the operations may be performed by a group of computers(as examples of machines including processors), with these operationsbeing accessible via a network (e.g., the Internet) and via one or moreappropriate interfaces (e.g., an API).

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but deployed across a number of machines. In some example embodiments,the one or more processors or processor-implemented modules may belocated in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Some portions of the subject matter discussed herein may be presented interms of algorithms or symbolic representations of operations on datastored as bits or binary digital signals within a machine memory (e.g.,a computer memory). Such algorithms or symbolic representations areexamples of techniques used by those of ordinary skill in the dataprocessing arts to convey the substance of their work to others skilledin the art. As used herein, an “algorithm” is a self-consistent sequenceof operations or similar processing leading to a desired result. In thiscontext, algorithms and operations involve physical manipulation ofphysical quantities. Typically, but not necessarily, such quantities maytake the form of electrical, magnetic, or optical signals capable ofbeing stored, accessed, transferred, combined, compared, or otherwisemanipulated by a machine. It is convenient at times, principally forreasons of common usage, to refer to such signals using words such as“data,” “content,” “bits,” “values,” “elements,” “symbols,”“characters,” “terms,” “numbers,” “numerals,” or the like. These words,however, are merely convenient labels and are to be associated withappropriate physical quantities.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or any suitable combination thereof), registers, orother machine components that receive, store, transmit, or displayinformation. Furthermore, unless specifically stated otherwise, theterms “a” or “an” are herein used, as is common in patent documents, toinclude one or more than one instance. Finally, as used herein, theconjunction “or” refers to a non-exclusive “or,” unless specificallystated otherwise.

Of course, the functions described herein for any single machine,database, or device may be subdivided among multiple machines,databases, or devices. As used herein, a “database” is a data storageresource and may store data structured as a text file, a table, aspreadsheet, a relational database (e.g., an object-relationaldatabase), a triple store, a hierarchical data store, or any suitablecombination thereof.

The invention claimed is:
 1. A computer-implemented method forprocessing an image comprising: accessing the image, wherein the imageincludes a first blurred region and a second blurred region; determininga positioning of a first blur kernel with respect to the first blurredregion; determining a positioning of a second blur kernel with respectto the second blurred region based on the positioning of the first blurkernel; and deblurring the image based on the positioning of the firstblur kernel and the positioning of the second blur kernel.
 2. The methodof claim 1 further comprising: determining a latent image based on theimage; generating the first blur kernel based on the latent image; andgenerating the second blur kernel based on the latent image.
 3. Themethod of claim 1 further comprising: receiving an input from a userthat specifies the first blurred region and the second blurred region.4. The method of claim 1 further comprising: determining a region sizefor each of a plurality of regions of the image based on an input blurkernel size, wherein the plurality of regions includes the first blurredregion and the second blurred region; determining a number of edgeorientations within each of the plurality of regions; selecting thefirst blurred region for deblurring based on the number of edgeorientations within the first blurred region; and selecting the secondblurred region for deblurring based on the number of edge orientationswithin the second blurred.
 5. The method of claim 1 further comprising:determining the positioning of the first blur kernel based on aproximity to at least one of a center of the image or a focal point ofthe image; determining a shift from the positioning of the first blurkernel based on a correlation between the first blur kernel and thesecond blur kernel; determining the positioning of the second blurkernel based on the shift from the positioning if the first blur kernel.6. The method of claim 1 further comprising: generating a deconvolutionof the first blurred region based on the positioning of the first blurkernel; generating a deconvolution of the second blurred region based onthe positioning of the second blur kernel; for an overlapping regionthat is included in both the first blurred region and the second blurredregion, generating a blending of the deconvolution of the first blurredregion and the deconvolution of the second blurred region; anddeblurring the image based on the deconvolution of the first blurredregion, the deconvolution of the second blurred region, and the blendingof the deconvolution of the first blurred region and the deconvolutionof the second blurred region. determining the positioning of the secondblur kernel based on the shift from the positioning if the first blurkernel.
 7. The method of claim 1 further comprising: determining a firstsize of the first blur kernel based on an autocorrelation of the imagewith a derivative of the image; determining a second size of the firstblur kernel based on a latent image of the image; determining asuggested size of the first blur kernel based on the first size and thesecond size of the first blur kernel; and generating the first blurkernel based on the suggested size of the first blur kernel.
 8. One ormore non-transitory computer-readable media having instructions storedthereon, which, when executed by one or more processors of a computingsystem cause the computing system to perform actions comprising:accessing an image that includes a first blurred region and a secondblurred region; steps for generating a first blur kernel for the firstblurred region; and steps for generating a second blur kernel for thesecond blurred region; and steps for deblurring the image based on thefirst blur kernel and the second blur kernel.
 9. The computer-readablemedia of claim 8, wherein the actions further comprise: steps forpositioning the first blur kernel with respect to the first blurredregion; and steps for positioning the second blur kernel with respect tothe second blurred region.
 10. The computer-readable media of claim 8,wherein the steps for deblurring the image include: generating adeconvolution of the first blurred region based on a positioning of thefirst blur kernel proximate to the first blurred region; generating adeconvolution of the second blurred region based on a positioning of thesecond blur kernel proximate to the second blurred region; andgenerating a deblurred image based on a blending of the deconvolution ofthe first blurred region and the deconvolution of the second blurredregion.
 11. The computer-readable media of claim 8, wherein the stepsfor generating the first blur kernel include: steps for determining afirst size of the first blur kernel based on an autocorrelation of theimage with a derivative of the image; steps for determining a secondsize of the first blur kernel based on a latent image of the image;steps for determining a suggested size of the first blur kernel based onthe first size and the second size of the first blur kernel; andgenerating the first blur kernel based on the suggested size of thefirst blur kernel.
 12. The computer-readable media of claim 11, whereinthe steps for determining the first size of the first blur kernelinclude: steps for determining a derivative of the first blurred region;steps for determining an autocorrelation between the first blurredregion and the derivative of the first blurred region; steps forapplying a filter to the autocorrelation between the first blurredregion and the derivative of the first blurred region; steps foridentifying one or more connected components within the autocorrelationbetween the first blurred region and the derivative of the first blurredregion; and determining the first size of the first blur kernel based onthe identified connected components.
 13. The computer-readable media ofclaim 11, wherein the steps for determining the second size of the firstblur kernel include: steps for determining a latent image of the firstblurred region; steps for estimating another blur kernel based on thelatent image of the first blurred region; steps for determining ameasurement of one or more connected components in the other blurkernel; and determining the second size of the first blur kernel basedon the measurement of the one or more connected components in the otherblur kernel.
 14. The computer-readable media of claim 8, wherein theactions further comprise: steps for determining a region size for eachof a plurality of regions of the image based on an input blur kernelsize, wherein the plurality of regions includes the first blurred regionand the second blurred region; steps for determining a number of edgeorientations within each of the plurality of regions; steps forselecting the first blurred region for deblurring based on the number ofedge orientations the first blurred region; and steps for selecting thesecond blurred region for deblurring based on the number of edgeorientations within the second blurred region.
 15. A computing systemcomprising: one or more processors; and memory, coupled with the one ormore processors, having instructions stored thereon, which, whenexecuted by the one or more processors perform actions comprising:receiving an image; determining a region size for each of a plurality ofregions of the received image based on an input blur kernel size;determining a number of edge orientations within each of the pluralityof regions; selecting a first region of the plurality of regions fordeblurring based on the number of edge orientations within the firstregion; and debluring the selected first region based on at least adeconvolution of the selected first region.
 16. The computing system ofclaim 15, wherein the actions further comprise: generating a first blurkernel for the selected first region; and debluring the selected firstregion based on at least the deconvolution of the selected first regionemploying the first blur kernel.
 17. The computing system of claim 16,wherein the actions further comprise: selecting a second region of theplurality of regions for deblurring based on the number of edgeorientations within the second region; positioning a second blur kernelbased on a position of the first blur kernel; and deblurring theselected second region based on at least a deconvolution of the selectedsecond region employing the positioned second blur kernel.
 18. Thecomputing system of claim 15, wherein the actions further comprise:determining a first size of a first blur kernel based on anautocorrelation of the image with a derivative of the image; determininga second size of the first blur kernel based on a latent image of theimage; determining a suggested size of the first blur kernel based onthe first size and the second size of the first blur kernel; generatingthe first blur kernel based on the suggested size of the first blurkernel; and deblurring the selected first region by employing the firstblur kernel.
 19. The computing system of claim 15, wherein the actionsfurther comprise: determining a derivative of the image; determining anautocorrelation between the image and the derivative of the image;applying a filter to the autocorrelation between the image and thederivative of the image; identifying one or more connected componentswithin the autocorrelation between the image and the derivative of theimage; and determining a size of a first blur kernel based on at leastthe identified connected components; generating the first blur kernelbased on the determined size of the first blur kernel; and deblurringthe selected first region based on the determined size of the first blurkernel.
 20. The computing system of claim 15, wherein the actionsfurther comprise: determining a latent image of the image; estimating asecond blur kernel based on the latent image of the image; determining ameasurement of one or more connected components in the second blurkernel; and determining a size of a first blur kernel based on themeasurement of the one or more connected components in the second blurkernel; and deblurring the selected first region based on the determinedsize of the first blur kernel.